File API Design Decisions

Learn what design decisions will best suit the file API.

In this lesson, we examine the overall operation of the file API and the interaction of the individual components. This will help us make certain design decisions while diving deeper into the design challenges.

Design overview#

There are many architectures that lead to good API design, but we can’t simply pick one and start developing. API design is often influenced by business requirements, technologies used, etc. Therefore, defining a simple step-by-step guide for choosing an architecture may not be possible. Often, a combination of multiple architectures work in tandem to achieve organizational goals. Let’s go over the design considerations of a file API and determine the suitable ingredients (architecture style, data format, etc.) to satisfy our needs.

The following diagram contains the detailed workflow of the components and services involved in the file API design.

Detailed workflow of a file API
Detailed workflow of a file API

Let’s look at the functionality of these components and services in the table below.

Components and Services Details

Component or Service

Details


File servers

  • Accept and process file API requests forwarded by the API gateway
  • Split requests into metadata (user and file information) and file content

Processing server

  • Performs encoding/decoding of data into different encoding schemes
  • Encrypt/decrypt data at-rest to prevent unauthorized access

User to file mapping system (UFMS)

  • Maps users to files and where those files can be found in storage

Blob storage

  • Stores binary data objects (file content)

Temporary storage

  • Stores files and objects temporarily before processing

User server

  • Handles user related requests forwarded by the API gateway

SQL database

  • Stores information related to files and users (metadata)

CDN servers

  • Geographically distributed network for serving data to the user from a nearby location instead of serving it from the blob storage


API gateway

  • Forwards client requests to the appropriate application server
  • Authenticates and authorizes client requests
  • Rate-limits client requests to keep the server from overburdening

Client

  • The consumer of the API service
  • Can perform lossless compression before sending a file

Workflow#

For a better understanding, we can divide the workflow shown in the illustration above into the following two parts.

  • Client to API gateway: Before performing any operation (upload, download, and so on), the API needs to authenticate the client and ensure that it has read/write access to the data. After successful verification, a user can upload, download, list, or delete a file from the server through HTTP requests.

  • API gateway to downstream services: As the request reaches the API gateway, the gateway forwards the request to the appropriate API services. If the request is to upload a file, the application server extracts the metadata (such as user ID, file name, file size, file location, etc.) from the incoming request and sends it to the user to file mapping system (UFMS), which adds it in a relational (SQL) database. The actual content of the file is stored in temporary storage. This data is further processed (compressed, encrypted, and so on) and stored in the blob storage using a storage-optimal format.
    Likewise, to download a file, the client requests the API gateway, which forwards its request to the appropriate application server that asks the UFMS for relevant information. After the server receives the required information, it retrieves the specified file contents, compiles the response, and returns it to the user. The server can also refer to the file contents present on the regionally distributed CDN servers for fast delivery.

Note: After data is successfully stored in blob storage, multiple copies of it are made and sent to different replica storage spaces (backup servers and CDN servers) for faster delivery, reliability, and disaster management.

Point to Ponder

Question

Why don’t we store the metadata in the same database as the actual file content?

Hide Answer

Metadata is being shared with many other services (such as a user service, an authentication and authorization service, etc.), and storing it in blob storage may not be a good idea because we often need to run many queries on it. Moreover, the actual file content is not being used by the API service and can be encrypted to store in permanent blob storage. Apart from that, having the same server do multiple tasks is not good for scalability. Finally, a failure of one service will not result in the failure of another if metadata and file content are stored separately.

Design considerations#

Now that we have a solid understanding of how the file API service works, let's go a step further and decide on the architectural style, data format, and HTTP version that we will use in our API.

Architectural style#

Let's consider the architectural styles we discussed in this course and see which works best for our file service:

  • REST: This provides a well-defined, standardized way to manage resources. Our API performs the most basic operations on a single resource (file), and REST can be a potential candidate because it provides this functionality well.

  • gRPC: This is best suited for communication-intensive systems because it supports the management of multiple request-response communication links (subchannels) over a single HTTP/2.0 connection (channel). It requires persistent connections and maintains connection states at each end to reduce the TCP handshake overhead. These features may not be fully utilized in our file API because individual client uploads may not be as intensive, and clients may not send multiple requests on a consistent network. Furthermore, gRPC is an action-oriented architectural style, which may add additional complexity to our design.

  • GraphQL: This excels at managing different resources with a single request from multiple endpoints. It also efficiently retrieves the required data to address overfetching/underfetching in RESTful APIs. However, interacting and retrieving data from multiple resources is not a use case for our service.

The discussion above suggests we should use REST because it has better support for different operations in our file API and avoids unnecessary complexity. Therefore, we propose REST as a communication protocol from client to API gateway and from API gateway to downstream services.

The choice of architectural styles for communication between client-gateway and gateway-downstream services
The choice of architectural styles for communication between client-gateway and gateway-downstream services

Data formats#

Let's consider the data formats we discussed in this course and see which suits our file service requirements the most:

  • JSON: This is human readable, compact, and easy to implement, with a built-in JavaScript (the most widely used scripting language for web services) support parser. We need to send user information, file information, and file data between the client (web application) and the server. For file and user-related metadata, we need readability for third-party developers to easily integrate our API into their applications. JSON meets all these requirements, making it a suitable format for our API.

  • XML: This also supports object representations and can represent more complex data than JSON, but it has some major drawbacks, such as larger encoded file sizes and a lack of built-in support for parsers, which makes it less suitable for our API service.

  • Binary: These formats are fast to process and compact in size compared to textual representations, but these formats sacrifice human readability, making it difficult to debug problems and understand errors. Our API needs human interactivity for client-server interaction and debugging issues. However, we can use binary formats to transfer, store, and retrieve actual file content.

As mentioned above, JSON provides everything we need and is also suitable for the RESTful API style. So, JSON is the data exchange format from client to API gateway and from API gateway to downstream services.

Remember: The actual file content is transmitted in binary format. Only the requests and responses are formatted in JSON for processing.

HTTP version#

We are using HTTP upload requests to transfer files between the client and the web server (more on this in the next lesson). Let's discuss which HTTP version we should use as the communication protocol in the file service and why.

  • HTTP/1.1: This supports range/chunked data transfers, using TCP and TLS for reliable and secure transferring. It also has the advantage of being supported by existing infrastructure. Our main focus is fast, secure, and reliable data transfers between client and server machines. HTTP/1.1 fits our requirements very well, making it a suitable option for a file API.

  • HTTP/2.0: This supports several additional features, primarily multiplexing streams for faster bidirectional communication over a single connection. However, sending a single file does not show significant improvement in performance as compared to HTTP/1.1, as shown by the following illustration.
    From the illustration below, we can see that there is no significant difference in the upload process of a single file. HTTP/2.0 seems to excel when it comes to uploading multiple files simultaneously. It also overcomes the head of line (HOL) blocking issue of HTTP/1.1. HTTP/2.0 can be considered for future work when our service adds the multifile upload feature.

Single file transfer in HTTP/1.1 vs. HTTP/2.0
Single file transfer in HTTP/1.1 vs. HTTP/2.0
  • HTTP/3.0: This was released very recently and is gaining quite a bit of traction in the industry. However, HTTP/3.0 has low infrastructure support, which increases development costs and narrows the range of supported clients. So, using HTTP/3.0 for slight improvements with added costs may not be a good choice.

A chunk is a small portion of bytes in the entire data sequence (complete file). In HTTP/1.1, we can enable chunked data transfers by specifying Transfer-Encoding: chunked in the request header. On the other hand, in HTTP/2.0, all requests are sent in byte ranges (chunk in HTTP/1.1) by default, and these ranges are called frames.

From the explanation above, we conclude that HTTP/2.0 supports segmented data transfers (called frames) by default, whereas in HTTP/1.1, we have to enable segmented transmission of data, referred to as chunks.

Considering the information above, HTTP/1.1 is a suitable option that can meet all our functional needs. Also, upgrading from HTTP/1.1 to HTTP/2.0 only requires clients and servers to support HTTP/2.0 connections, while the rest of the application logic can remain the same. We are adding support for both HTTP/1.1 and HTTP/2.0 to our file API. However, our discussion will mainly focus on HTTP/1.1, which is the minimum requirement for our API to function. We will also explore possible cases where HTTP/2.0 outperforms HTTP/1.1.

Point to Ponder

Question

Why don’t we use File Transfer Protocol (FTP) requests to transfer data between the client and server?

Hide Answer

FTP, by default, transmits data in plaintext, and it’s less suitable for frequent requests to upload small files. On the other hand, HTTP works well with interrupted requests (by asking for a specific byte number in the file instead of starting afresh) and firewalls, making it suitable for small file uploads.

Summary#

A summary of our design decisions is given in the table below:

Design Consideration

Client-to-API Gateway

API Gateway-to-Downstream Services

Architectural Style

REST

REST

Data Format

Binary for files, JSON for everything else

Binary for files, JSON for everything else

HTTP Version

HTTP/1.1

HTTP/1.1

Requirements of the File API

API Model for File Service